Deep learning sometimes appears to work in unexpected ways.

Most of this work was done when Fanghui was at KULeuven.

This important phenomenon commonly appears in implemented neural network architectures, and also seems to emerge in epoch-wise curves during the training process.

This paper demonstrates interesting new phenomena that suggest that our understanding of the relationship between the number ofdata points, the number ofparameters, and the generalization errorisincomplete,evenforsimplelinearmodels.

We show that the generalization curve can have an arbitrary number of peaks, and moreover, locations of those peaks can be explicitly controlled.